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Invention Disclosure Form Supplement 

1. NAMES 

Christoph A. Aktas, John W. Yates, Phillip C. Meredith 

2. TIUI 

A subsystem for device and media sensitive access and processing of multimedia 
messages 

3. PURPOSE AND PROBABLE FIELD OF USE OF THE INVENTION 

The purpose of this Invention Is to provide a subscriber or a user of a unified messaging mailbox 
with efficient intelligent media- and device- sensitive methods to access and process (eg., read, 
listen, forward, and search) messages. The invention Introduces information summarization and 
media conversion capabilities to selectively treat multi-media messages and message 
attachments so that they can be efficiently handled by mobfle devices like PDAs (Personal Digital 
Assistants), pagers, or phone devices (with or without a text display feature). Furthermore, the 
Invention introduces message content analysis capabilities that wilt recognize linguistic 
relationships between messages regardless of the media type. The invention also describes the 
ability to present these linguistic relationships as a "graph" along with the standard messaging 
relationships (Message arrival time, subject, sender, etc.) And, finally, the invention also 
introduces a message referencing option that allows simpler message selection from certain 
devices. 



4. PLANNED USE IN PRODUCTS 

The system could be potentially Integrated in all our messaging, mobility, and collaboration 
related products including Xpressions. Specific clients could be developed as separate products 
and work with any existing messaging system. The capabilities could also be integrated into call 
center-based e-mail communication systems for interaction with customers. New virtual agent 
products could utilize these capabilities. 

5. ABSTRACT 

This invention describes capabilities that will, especially when used in coordination with each 
other and existing capabilities, increase the efficiency of unified maSbox users. Media 
conversion, information summarization, sophisticated data relationship identification, data 
relationship presentation and message selection tools are described that allow the unified matibox 
user organized and efficient access to his/her messages, customized to the device they are using. 
With these capabilities, the problems of overwhelming amounts of data in a variety of media 
(often incompatible with the user's current display device) will be reduced. 



6. BACKGROUND INFORMATION 

ft 1 What is tfre problem solved by your invention? 



Witness: .. 
Date: 
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A unified mailbox where all kind© of medc© (voioa, few. e-mail, and video) atro mads accessilbte 
and/or visibte from virtually anywfws to a subscriber or use? in one basket is a convenient means 
o? communication when compared so handling multiple maibosea with distinct media Current 
solutions for a unified mailbox ssre inefficient however, for someone with an intense 
eemmunfeate style &Jtd a frequent ncsd to handle his/her messages remotely. The mismatch 
a? m£d& fyp£ of th© information and the capabilities of the various (often Smited) dovtes used for 
remote access places a heavy burden on the usetr and Gft© intorfeco of the sysSem* This is 
especially trus for tfie Intefteoes utfllging a telephone with rto display, or handheld devleas with 
limited display capabilities. 

Some of the probEems aris© in the context of compound and/or tercgthy messages In connectfon 
with one or ih® othor socsss m®@ns. For examples: 

o Haw t© handle voice md\ fax attachments from a next-only e-mail capable dsvcee? 
o How to fasat bfftgthy o-msils from a voice-cnly interface or text-interface utith limited 

capabilrtfes. Ev©n tarh^n the device has a fuflty fijncBonsO GUI rnterfec©, thsns te room tor 

increased (stfftG&arcey with lar@§ amounts o? data 
o Hnw Bo efficiently p^&snt fih© information in various ofScoe document fonnsta (e.g., Word! 

Processor, Spsigbd&heet, and Presenilations) assascated wKfo © mes§®g©. 
o How to beats amd visually present related nrtesssg©© and attachments? 
o Ko^ to sasily rof&n&ftce messages in the message $xm? 

Other problems arfsa dye go tfo$ inepsssed ©mounfc of Information the u nffied maiibo» eara provide. 
Cunnent mgefeaiftfsms far organising @nd presen8rtg relationships among massages (Elstlirtg by 
arrival time, subject, sender, etc.) ana Insufficient for a Barge number of messages of ve?yfetg> 
medSa (and, especiaGy. imtoed media within a giv@n massage). The user natures a ftelble, 
media independent way of finding and navigating related messages. With currant systems, for 
example, the user Is unabte to rocogntes that there cs a relationship be&seen a v©5c© message 
and a tec w&thouft HsfcsnEng to the message and dtaptayta^prtnang the flax. This Invention 
proposes ©tat the system convert these messages to feariL enafyzQ the masutts and present any 
ralaEonships diseowsrad. 

Finally, bscaus© the presentation of usnflfied maSbOK ^formation is mom complex, espsdalliy f? 
relationships d$$crihsd in the previous paragraph sire EncorporatBd into the presentation, 
identifying an individual mm (message or message sfechinrent) for ftjffihsr action can become 
probtemate. Mow does the cleent/user identify to the server which msss&go is to acted upon? 
Are entire massage and Its attachments to ba involvad? Is it a single attachment or only the 
orig5n©3 msssag© body? And if the message© are prosemted in a "graph* format, how do@$ the 
ussr soloct an individual fern? This invention also addresses th& prob&m. 



6.2 VW 5 ?^ techniques prior to your invention w®m itsod to perform the function of your 
invention? 

Current tschnotogy that attempts to address these problems varies depending on the user's 
device: 

1 . Current systems offer media sensitivity for message retrieval wh@n provided with a graphical 
user interface (GUI) from a PC client or Web. If © particular media or office document 
attached to an e-mail, the user needs to ctick-on it in ostier to launch a specie client (for 
instance: audio player for voics, tiff-viewer for fex, video player to vlet& a video m&ssage. 



Witness: 
Date:...! 
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Word to a view or print MS-W6rd document) to listen, to view, or to print the document 

For users with Intense communication requirements (e.g. executives or customer service 
agents who receive hundreds of compound messages deity) there are no means to quickly 
process/read rnbox messages except by the sender information, the subject line, and maybe 
few fines of the message body. In order to read messages, the user has to dick on or mark a 
certain item in a graphical interface rn order to get to the message body. 

No content summarization of lengthy text messages or respective attachments is available 
yet that would remarkably improve the efficiency of handling the daily information avalanche 
in the office. 

Search is provided in certain clients on e-mail onty, but ft does not provide visual display of 
content and temporal relationships. No search capability exists yet for non-text messages. 

No powerful message referencing capabilities are offered to efficiently handle multimedia 
messages from remote devices (e.g. telephone). 

2. If a unified mailbox is accessed from a telephone Interface, voice and e-mail messages are 
retrievable and the user can listen to both; here, text-to-speech technology provides a means 
to convert the e-mail to voice. A fax message can be forwarded to a lax machine or printer. 
However, if an e-mail contains a voice attachment the systems are able to indicate that, but 
are unable to access its content Similarly, the contents of a fax or other documents attached 
to an e-mail are indicated but not accessible to the user using the telephone interface. 

If an e-maH is lengthy, based on the interface provided, the user maybe able navigate through 
ft buy accelerating its reading, skipping parts etc. in order to listen to it completely. There is no 
means of text content summarization applied to shorten the process without eventually 
fosingfeWppJng critical content 

3. If messages are forwarded to a handheld device via a wireless service but the device has 
limited text-display capabilities only certain pans of the email (From, Subject and a limited 
number of characters of the message body) can be displayed. 

If the critical information in the message is not in the beginning of the message body that Is 
displayed, it is "losr to the recipient He/she has to use other access methods (like Web) or 
make a call into the messaging system/server to retrieve the full text message (by Bstenrng to 
it or by initiating a printing to fax device nearby). 

Importantly, voice and other media attachments are indicated but not transmitted and/or 
displayed on a text-only display. The user needs to use other access methods (Wfcb and/or 
Telephone) to retrieve the messages. Additionally, no text content summarization methods 
are utilized to deal with access device technology limitations. 

6.3 What am the disadvantages of these prior techniques? 

FuU message sensitivity is provided within a GUI-only (desktop PC client or Web). However even 
GUI interfaces lack any means to summarize message content in order to make it more efficient 
to the recipient to read his/her lengthy messages. Also, there are yet no means to summarize 
content of attached documents. 



Date: 



Witness: . 
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Within the telephony interface, the media and device sensitivity is limited to voice and e»ma& 
Again, no techniques of text content summarization are applied yet tn order to make the retrieval 
of the message information over the phone more convenient 

in case of handheld or mobile devices with limited text-display capabilities, the problem is that 
lengthy messages are not transmitted usually by the wketess/paging service providers. 
Additionally, any other media attachments are lost*. 

No content summarization of lengthy text messages or respective attachments is available yet 
that would remarkably improve the efficiency of handling the daily information avalanche in the 
office. 

Search is provided in certain clients on e-mail only, but it does not provide visual display of 
content and temporal relationships. No search capability exists yet for non-text messages. 

No powerful message referencing capabilities me offered to efficiently handle multimedia 
messages from remote devices (e.g. telephone), 



6.4 What am the advantages of your invention over the prior techniques? 
The proposed invention solves the problems above by utilizing advanced media conversion 
methods, analysis and summarization of message content and intelligent forwarding concepts. 
Principally It provides access device and media sensitive intelligence for a mailbox when 
retrieving or forwarding a particular message. 

The proposed techniques provide the following advantages: 

a) Advanced media conversion - the concept of media conversion is extended beyond text-to- 
speech to other attachments; a speaker-independent, large vocabulary, telephony-quality 
speech recognition engine is utilized to convert a voice message to text or to convert the 
voice track of a video attachment into readable text Similarly, tax information is converted 
into text Use of this feature will allow users on devices unable to handle the original media 
type of a message to present the message to the user. 

b) Content summarization provides increased efficiency in message handling - the 
summarization of a message content is an improvement toward efficiency, even in a GUI 
environment In a GUI environment lets say in Microsoft Outlook, the user could point to the 
item in the Irt-box to have displayed in small pop-up box the summary of that message. But its 
benefits are obvious in case of a forwarded lengthy message to a handheld device with 
limited display capabffrties. The same Is true for reading a message over the phone. 
Summarization applied to attached media (e,g. fax. Word document) extends even the media 
content accessible. 

c) Media and device sensitivity- Both, the media conversion and the content summarization 
applied together provide compatibifrty with the access device. Depending on the user, the 
types of potential access devices are usually predefined; therefore messages along with their 
attachments that form the message content can be tailored to those devices while accessed 
or forwarded according to a profile. This ensures the availability of more information to the 
recipient at the device of choice and that is probably most convenient Still, if the user 
requires more information, he/she can utilize another access method (telephone or Web). 



Date: „} 
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d) CfQ$&M$dra Search end Visual Display -Often messages related to 3 specific topic of 
interest to the user are in different media and spre&d throughout the massage stare (e.g, 
different Elders). The cross-media search would find these messages aind present them to 
th© usar In a way thst makes the cefftfetnt and time relationships dear- altering ©(FKcfemt use 
of the otherwise overwhelming amounfc of information. Ultimately , the search can utilize 
sophisticated HngulsticaJly based analysis toote to discover tfiie message retstionsfiips. 

e) Simpler M&ssog® Refepancing - Additional!^, a ref&renee number seherrta to massages is 
proposed. AGI messages 5n a particular group o? messages of interest! to the us©r utfoutd be 
assigned si reference number to to ussd In farmer actions. Thus o PDA usor cart, ibr 
example, get a summary of messages with referee© numbers and an indicate off the 
f?resss§2 type. This reference number could then be used to access that message, end 
through ft, a particular attachment of that message tor further action (Print the first attachment 
of messsg© 7 to fax; play message 3 on tetepfton©). Ultimately, voice commands could be 
us©d to onvok© action© on items mora efficiently ("Fax Item 12345 to my hpteP) 

0 Qombmatkxm of the above features ~ PeweriW user interface functions can b© built from 
thesafaatess. The ssmpte scenario bsrfew shorn on© possibility. Many others are possible. 



?. DETAILED E)ES€^lira©W 

7, 1 D&ailod structural and functional opomtim 

The base structural components requiired ibr th3 proposed fieaftjfi©© ©pq: 

o a mjlflrredla PC wRh a client program fchat provides a muftimsdia messsg® inbox. 
Aflterraaitlvelfy, a server that proves muftiimedia message Inbox tor several users on © 
ne&tfertc 

o a subsystem that detects med& attachment© in messages in a msilba*. 

o A subsystem fihaft converts msdfe attachments into another media type using te&fo^spesch, 

fett-to-text video vc5c© track into teat and ultimately speech-to-text 
o A subsystem that analyzes and summarizes text content of origins! or converted msdia in 

rsspsct of tfi© linguistic meaning 
o A subsystem tftst pushes appropriate media eseoridins to en access d^vkse and) message 

purpose, S3 defined in a profit 
o A subsystem that identifes csioss-msdia Entem^tronships between messes© and coMrote 

the maifla conversions necessary for this analysis 
o A subsystem that controls th© reference scheme. 



The following describes an example of this process invofvrng a user mat has a mul^nedia 
maftbox and a data pager who receh/es a Voice Message. The problem Is to provide me "basf 
information to fr@ pager so fte user csn proceed moss efifefenfcfy. What is me "besf Information 
will vary according to the user's aefcjal preferences, but will most lively indudo sender 
identification and meaningful portions of (ha massage Itself. In addition, there are probably 
messages the user would prefer to dstey any handMng of until an appropriate device is available. 
Thus the steps would include: a) ftftertng messages to be processed, b) speech-to-testf 
conversion, c) summarizotton and post filtering, and d) s©Ssction and de\t^ery of this information to 
th© devic©. 



Wetness: 
Date: ....s 
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As the resources involved in processing a message may be large. Speech-to-Text b expensive 
in Its use of resources. Interrupting the user with aH but the most Important messages can be an 
unnecessary expense of the user's time and attention. Thus a mechanism to prevent the 
presentation of a message to a given device is important This filtering is currently available on 
products like Microsoft Exchange/Outlook that can filter based on a variety of data including 
sender, message priority, etc. 

Speech-To-Text Conversion 

VWh a speaker/microphone independent speech-to-text engine, the selected Voice Message then 
can be converted to text 1 . Thte would be most efficiently accomplished on the server side, 
perhaps with a dedicated "helper" server explicitly tor me server so as not to disturb other 
processing on the server. The resulting text massage would then be associated with the original 
message (as the text message body or as a separate attachment). • 

Post-Conversion Filtering and Summarization 

Another filtering step might then be appropriate, preventing processing of messages that appear 
not to be on a topic deemed important to the user. If ft does not appear important, it would then 
remain in the mailbox to be processed. 

If the message survives the filtering step, the text would then be summarized. Most simply, 
summarization would include reduction to a list of keywords and phrases found within the text 
The summarization would be created by removing from the message words/phrases not found 
within the user-defined Hst erf keywords/phrases. More complex summarization would include 
allowing the user to specify the keyword/phrase list based on the sender of the message'. The 
most complex summarization method would involve sophisticated grammatical parsing and 
analysis. 

Data Selection 

For notification via data pager, the user would configure what part of the filtered summarized 
message they wish to be sent The data available for selection would include Sender Name, 
Time, Summary, Message Priority and un-summarized Text (and other fields as available). The 
user would describe a template that indicates the information desired and the number of 
characters of each field desired. For example 3 : 

From %SJSND£ft% at %TIMB%: %X0OSUWARY* 

WbuH indicate the user wants a string that includes the entire sender name, the received time 
and the first 100 characters of the summary to appear on his pager. 

When the user receives the page, the summary information should give him/her enough 
information to determine how critical the message is. If it appears critical, he/she could get to a 
more appropriate device (e g a telephone) and listen to the full message. 

1 Interim solutions for non-speaker/microphonc independent engines wilt be discussed in a separate 
document (maybe in section 7.2), 

2 Since the message will be a speech-to-text conversion, the keywords AND THEIR HOMONYMS should 
be checked. Aa option on the summarization, like a check box that says "Allow homonyms" would seem 
the best way to handle this. 

* The exact template format is to be determined; this is just to describe the concept 

Witness; .^ZC&e ^&f. . ... 
Date: ....^^^^^^^^^^^^^HT 
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7. 2 Are there attemativB methods or different structural embodiments of your invention? Cen 
the general idea or technique of your invention be extended to other related fields? 

a) The summarization concept can be appBed in context of any other handheld device that is not 
limited to text only (eg. Windows and Windows/CE devices) 

b) A morphing process for a certain message in context of any particular target device can be 
defined. The morphing process would be a combination of message filtering, message 
restructuring, data conversion, data summarization, data selection and notification steps that 
are configured to handle particular media types for particular target devices. Each user could 
define a set of rules and parameters for each device type defined would determine how the 
message is morphed. For example, a user could have a Voice Message-to-Pager morph 
definition that would do the following: 

1 . Fitter based on sender and priority, removing from further processing (le. leaving on the 
server) message that are not deemed urgent enough to disturb the user while out of the 
office. 

2. The message would go through speech-to-text conversion and the resulting text stored back 
in the message. 

3. The text would be summarized based on criteria defined by the user. 

4. Another Filtering step, based on the summarized/converted text would then weed out items 
whose content makes them less urgent 

5. Data fields from the message (tike sender, priority, summary and text) are selected and a 
notification message created. 

6* The message is sent to the pager as a notification. 

In general, a morphing process win include these steps in some order determined by the user in 
addition, message restructuring steps would allow the user to handle multiple attachments of 
varying media on the message. For example, the user could select that a summary of the 
attachments be created (attachment name and media type) - or could request that the 
attachments be expanded, converted and summarized as described for the single media 
message above. 

Examples 

• Voice Message to Data Pager (as I described already ) 

• Text Message to Phone 

1. A Filter out unwanted messages based on sender and priority. 

2. Summarize message to reduce its size. 

3. Convert text-to-speech. 

4. Play the converted message over the phone. 

• Fax-to-PDA 

1. Filter out unwanted messages based on sender and priority. 

2. Convert tax-to-text. 

3. Summarize. 

4. Select Data Fields 

5. Send the summary to the PDA and "Notify" the user. 
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7.3 Which features are betteved to be new? 

• Conversion of media applied to any media (voice, fiax, video) in a mailbox 

• Text summarization of original or converted media from a multimedia mailbox 

• Improved efficiency even in a graphical environment by summarizing the content of a 
message according to its linguistic meaning 

• Media and device sensitivity when accessing multimedia messages in a message inbox 

• Linguistically based search for relationships between message/message attachments of 
differing media types. 

• Integration of these capabilities into useful features for the mailbox user. 



Date: 



Witness: .... 
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